Skip to content

Recover from invalid values while deserializing - #178

Merged
xoofx merged 8 commits into
xoofx:mainfrom
StefanGreve:feat/invalid-value-handling
Nov 8, 2026
Merged

xoofx merged 8 commits into
xoofx:mainfrom
StefanGreve:feat/invalid-value-handling

Conversation

@StefanGreve

Copy link
Copy Markdown
Contributor

Why

When a converter rejects a value, deserialization stops at that value. For a document that a person edits by hand,
such as a configuration file or the front matter of a Markdown page, that is often the wrong trade: one malformed
date: costs the whole file, and the author learns about the second mistake only after fixing the first.

The cases I care about:

What

One option and two opt-in delegates on YamlSerializerOptions, working with reflection and source-generated contexts:

var options = new YamlSerializerOptions
{
    SourceName = path,
    InvalidValueHandling = YamlInvalidValueHandling.Skip,
    InvalidValueCallback = v => log($"{v.SourceName}({v.ValueStart.Line + 1},{v.ValueStart.Column + 1}): {v.Path}: {v.Message}"),
};
  • InvalidValueHandling selects what happens when a converter rejects a value:
    • FailFast (default): the first invalid value throws, exactly as today.
    • FailAtEnd: every recoverable value is skipped and the read continues. At the end the call throws one
      YamlInvalidValuesException listing all of them, located at the first.
    • Skip: every recoverable value is skipped and the call succeeds.
  • InvalidValueCallback runs once per skipped value, in document order. InvalidValuesFinalizer runs once per
    Deserialize/TryDeserialize call with every skipped value, only when there is at least one. Under FailAtEnd it
    runs before the exception is thrown and can throw to replace it. Neither runs under FailFast.
  • YamlInvalidValue is shaped like YamlUnmappedMember: SourceName, Path ($.servers[2].port), KeyStart,
    ValueStart, Name, TargetType, DeclaringType, Instance (null for constructor-bound types), Value
    (YamlElement?, captured only when a callback or finalizer is set), Message and the YamlException the converter
    threw.
  • A skipped member keeps its initializer value, or its existing instance under JsonObjectCreationHandling.Populate.
    A skipped constructor parameter keeps its default. The key still counts as present for required members and for
    duplicate detection.
  • YamlReader.CreateException(message) gives a custom converter a YamlException located at the current token. Thrown
    before the value is consumed, it is recoverable.
  • YamlException.Reason is the message without the source: (start) - (end): prefix. YamlInvalidValue.Message
    returns it, so a callback that formats the location itself does not print it twice.

Options stay immutable and shareable; all per-call state lives in the reader, in the same session that #165 added
for unmapped members.

Semantics

The mechanism follows Tomlyn, which wraps each member read in a filtered catch and skips the value when the reader
has not moved. Here:

  • Recovery sites are member values and constructor arguments, including those reached through merge keys, in the
    reflection path and in generated code. Collection elements, dictionary values, the root value and extension data are
    not recovered.
  • What is recoverable: only a YamlException that is not a parser error (SyntaxErrorException,
    SemanticErrorException), and only while the reader is still on the token where the read began, compared by token
    type and start offset. Everything else stays fatal: other exception types, which Wrap exceptions thrown while reading a value in YamlException #177 still wraps in a
    YamlException at the entry points, and every read that already consumed part of the value. A converter bug is
    therefore never reported as invalid content.
  • A fatal error after recoveries: under FailAtEnd the call throws YamlInvalidValuesException with the values
    recovered so far and the fatal error as InnerException. Under Skip the fatal error propagates unchanged. The
    finalizer does not run in either case. NotSupportedException and IOException keep propagating unchanged, as in
    Wrap exceptions thrown while reading a value in YamlException #177.
  • Unchanged errors: UnmappedMemberHandling.Disallow, DuplicateKeyHandling.Error, a missing required member and
    polymorphic discriminator errors, because none of them is thrown by a member value read. TryDeserialize returns
    false under FailAtEnd and true under Skip.
  • Inside a buffered polymorphic node the path is complete, but the marks are relative to the node, the caveat
    YamlUnmappedMember already documents.
  • Deserialization only. A write has no input to point at, and a converter that cannot write a value has a bug, not
    bad input.

One observable change outside FailFast: for a malformed value such as title: [Ten, the string converter's
"expected scalar" exception is recovered, the skip then hits the unterminated sequence, and the parser's
SemanticErrorException surfaces instead of a plain YamlException. FailFast is unchanged.

Performance

Under the default FailFast nothing is created: no session, no path tracker, and the catch filters return at the
option check. The try regions in generated code exist in every mode, so I measured the default path against main
(8fbc666) with the existing benchmarks. Allocations are identical to the byte and every time difference lies inside
one standard deviation.

Benchmark main branch Allocated (both)
SharpYaml_Deserialize_Poco 2,018 us (StdDev 64) 2,000 us (StdDev 46) 1948.39 KB
SharpYaml_Deserialize_Poco_UnknownKeys 2,079 us (StdDev 94) 2,030 us (StdDev 74) 1950.6 KB
SharpYaml_SourceGenerated_Deserialize 1,964 us (StdDev 121) 1,838 us (StdDev 35) 1872.89 KB
SharpYaml_SourceGenerated_Deserialize_MergeKeys 3.623 us (StdDev 0.13) 3.565 us (StdDev 0.08) 6.87 KB

With reporting enabled, the session and the path tracker from #165 are reused, and the value is captured as a
YamlElement only when a callback or finalizer consumes it.

API surface

  • YamlInvalidValueHandling (FailFast, FailAtEnd, Skip) and three properties on YamlSerializerOptions.
  • YamlInvalidValue (sealed) and YamlInvalidValuesException : YamlException, so catch (YamlException) and
    TryDeserialize keep working.
  • YamlReader.CreateException(string) for converters.
  • YamlException.Reason.
  • Two protected static members on YamlSerializerContext, CanRecoverInvalidValue and RecoverInvalidValue,
    because generated code lives in the user's assembly. They stay off the reader's public surface; the alternative
    would have been public reader members as in Add global unmapped member callback and finalizer #165.

Internally, YamlUnmappedMemberSession became YamlReportingSession, since it now carries both kinds of report.

The names follow Maven's --fail-fast and --fail-at-end; Skip matches UnmappedMemberHandling.Skip on the same
options object.

Follow-ups, not in this PR

  • Recovery for collection elements and dictionary values. A value that is itself an element, such as abc in
    Retries: [1, abc, 3], or the value of a dictionary entry still fails the call in every mode, because the list
    and dictionary converters read their items in a loop with no catch around each one. A member of an object inside
    a list is recovered, since that read goes through the object converter. I left elements out because skipping one
    has no obvious meaning: a member keeps its default, but dropping an element shifts the indices after it, a
    placeholder is impossible for a value type, and a dictionary entry could lose the entry or keep the key with a
    default. Tomlyn stops at members as well. If wanted, it is the same filtered catch around each item read in the
    collection converters and their generated equivalents, with "drop the item" as the rule.
  • Reporting without exceptions, which warnings that are not errors would need.
  • A generated context that lacks a reader for a member type throws YamlThrowHelper.ThrowNotSupported, a plain
    YamlException, before consuming anything, so under FailAtEnd and Skip that configuration error is reported as
    an invalid value. The generator throws NotSupportedException directly for its other configuration errors; aligning
    the helper would be a small separate change.
  • The built-in DateOnly converter calls DateOnly.TryParse with the invariant culture, so it accepts 09/13/2026,
    while System.Text.Json accepts only yyyy-MM-dd. Worth its own issue.

Written with Claude Code (Claude Fable 5.1).

StefanGreve and others added 8 commits November 8, 2026 16:26
* Introduce YamlInvalidValueHandling with FailFast as the default, so a
  converter rejection keeps throwing immediately unless a caller opts into
  FailAtEnd or Skip
* Model each skipped value as YamlInvalidValue and aggregate them in
  YamlInvalidValuesException, whose location is that of the first value and
  whose inner exception names a later fatal error
* Let the reporting session hold the invalid value delegates and collect
  values under FailAtEnd, and add YamlReader.CreateException so converters
  can reject a value at the current token before consuming it

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Wrap each member and constructor argument read in YamlObjectConverter with
  a filtered catch, so a YamlException thrown before the value was consumed
  skips the value under FailAtEnd and Skip instead of ending the read
* Compare the token type together with the start offset in the filter, since
  Mark has no equality and a mapping shares its offset with its first key
* Capture the skipped value as a YamlElement only when a callback or
  finalizer consumes it, keeping the FailAtEnd collection cheap

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Run the invalid value finalizer and throw YamlInvalidValuesException from
  the session's Complete, so a read that recovered values still fails under
  FailAtEnd while Skip lets it succeed
* Fold an error that ends the read into the aggregate as its inner exception
  in the four DeserializeCore overloads, so the values recovered before the
  failure are not lost with it
* Keep NotSupportedException and IOException outside the aggregation, as they
  report a configuration or source error rather than invalid content

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Wrap the matched branches of each generated ReadMember in one try/catch
  and set the target type per branch, so the catch covers value reads alone
  without emitting one try per member
* Expose CanRecoverInvalidValue and RecoverInvalidValue as protected static
  members of YamlSerializerContext, since generated code lives in the user's
  assembly and cannot reach the reader's internal helpers
* Reject SyntaxErrorException and SemanticErrorException in the filter, as a
  document the parser cannot read is not an invalid value and skipping the
  node would only throw again inside the handler

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Keep the message passed to each constructor and surface it as Reason, since
  Message bakes in the source name and marks and a caller that reports the
  location in its own format printed it twice
* Return Reason from YamlInvalidValue.Message, so a callback or finalizer
  gets the converter's wording alone while Exception still locates the value

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: CodeAlta <noreply@codealta.io>
@xoofx
xoofx merged commit dd3433f into xoofx:main Nov 8, 2026
@xoofx

xoofx commented Nov 8, 2026

Copy link
Copy Markdown
Owner

Thanks @StefanGreve, this is a very solid contribution! Merged via squash.

Review notes:

  • The recovery rule (a YamlException that isn't a parser error, while the reader is still on the starting token, checked by token type + offset) is conservative in the right way: a converter bug or a partly consumed value is never reported as invalid content.
  • The default FailFast path stays free: no session, no tracker, and the filters return at the option check. Generated code wraps one try per ReadMember rather than one per member, which is a good call.
  • Locally: build clean with no warnings, all tests pass in both reflection and generated modes, and lunet build has no warnings.

I pushed one small commit (c8faa6b) with a regression test for the documented guarantee that a skipped key still counts for duplicate detection, for both a member and a constructor parameter. The test is reflection-only. While checking it I found that generated contexts don't detect duplicate member keys at all, even with valid values and the default DuplicateKeyHandling.Error. That gap predates this PR, so I'll track it separately.

The follow-ups you listed (recovering collection elements and dictionary values, ThrowNotSupported vs NotSupportedException in generated contexts, DateOnly parsing leniency) all sound reasonable as separate issues.

This comment was written by CodeAlta.

@StefanGreve

Copy link
Copy Markdown
Contributor Author

@xoofx Sounds good to me, I will do the follow-up tasks listed in this PR sometime next week.

@StefanGreve
StefanGreve deleted the feat/invalid-value-handling branch November 8, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants